[FLINK-21164][rest] Delete temporary jars #14777
Conversation
|
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit c833d99 (Fri May 28 08:15:03 UTC 2021) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
There was a problem hiding this comment.
Thanks for creating this PR @zentol. The change itself looks good. I had a couple of comments concerning other places where the PackagedProgram is being used.
...k-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/JarPlanHandler.java
Outdated
Show resolved
Hide resolved
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/JarRunHandler.java
Outdated
Show resolved
Hide resolved
| @@ -103,6 +103,11 @@ public JarRunHandler( | |||
| executor) | |||
| .handle( | |||
| (jobIds, throwable) -> { | |||
| try { | |||
| program.deleteExtractedLibraries(); | |||
There was a problem hiding this comment.
Isn't JarListHandler also susceptible to the problem?
There was a problem hiding this comment.
I am also wondering whether the ApplicationClusterEntryPoint really cleans up a PackagedProgram after using. Maybe this is something to investigate as a follow up.
There was a problem hiding this comment.
Yes, the JarListHandler has the same problem. I did not know it actually creates a PackagedProgram.
As for the ApplicationClusterEntryPoint, the files should be cleaned up when the entry point shuts down because we call File#deleteOnExit in PackagedProgram#createTempFile.
There was a problem hiding this comment.
Now whether they are actually deleted depends a bit on when the JVM executes the delete calls. If the ClassLoader is still around at that time, then the deletion may in fact fail.
Explicitly closing the ClassLoader in the PackagedProgram (which unfortunately leaks into the JobGraph...) is a separate issue (see FLINK-9844).
There was a problem hiding this comment.
Alrighty. Thanks for the update.
…nitor/handlers/JarRunHandler.java Co-authored-by: Till Rohrmann <till.rohrmann@gmail.com>
…nitor/handlers/JarPlanHandler.java Co-authored-by: Till Rohrmann <till.rohrmann@gmail.com>
|
@tillrohrmann I updated the PR. I will also file a follow-up ticket to explicitly call close() in the entry points. |
Modifies the Jar(Run|Plan)Handler to call
PackagedProgram#deleteExtractedLibrariesonce we are done with the job submission / plan generation.This is not covered by tests; I don't see a good way to do that since from the outside it is neither obvious which files are supposed to (not) exist nor where the temporary files would be located (by virtue of PackagedProgram not respecting the
io.tmp.dirsoption).